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ABSTRACT 

These  Research  Notes  form  part  of  a  series  of  notes  extracted  from  work  undertaken  by 
Innovation  Science  in  the  establishment  of  Openness  and  Evolvability  assessment 
Methods  and  Processes.  This  set  of  Research  Notes  focusses  on  an  introduction  to  the 
assessment  of  Openness.  This  work  was  undertaken  from  the  late  1990s  to  2007  and 
focussed  on  the  application  to  Submarine  Combat  Systems. 
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1.  Introduction 


These  Research  Notes  have  been  extracted  from  work  undertaken  by  Innovation  Science 
under  contract  to  Defence  Science  and  Technology  Group  during  the  period  from  the  late 
1990s  until  early  2007. 

In  entirety  the  Research  Notes  form  a  subset  of  the  overall  assessment  Methodology  and 
Processes  developed  to  assess  system  level  Openness  and  Evolvability. 

The  Research  Notes  within  this  report  focus  on  an  introduction  to  the  assessment  of 
Openness. 


2.  What  are  Openness,  Evolvability  and  Open  Systems 

Openness:  The  extent  to  which  components  (third  party  and  integrator)  can  be 
independently  integrated,  removed  or  replaced  without  adverse  impact  on  the  existing 
system. 

Evolvability:  The  ease  with  which  a  system  or  component  can  be  modified  to  take 
advantage  of  new  software  or  hardware  technologies. 

Open  System:  A  collection  of  well-bounded,  interacting  components  (software,  hardware 
and/or  human)  with  well-defined,  published  and  configuration-managed  interfaces, 
interconnected  by  well-defined,  published  and  configuration-managed  infrastructures, 
and  where  sufficient  procedures  and  policies  govern  the  system  to  ensure  components  can 
be  independently  added,  removed  or  replaced  without  eroding  the  system  architecture. 


3.  Scope  Selection 

An  evaluation  of  openness  can  be  applied  at  many  levels.  The  focus  could  be  on  the  ability 
for  different  federated  systems  to  be  interconnected  to  form  a  system  of  systems.  Or, 
perhaps  the  focus  could  be  on  the  ability  for  different  vendors  to  be  able  to  provide  small 
functional  elements  within  a  single  application.  The  scope  of  the  openness  requirement 
will  alter  the  importance  of  different  business  and  technical  characteristics  that  combine  to 
achieve  an  open  solution. 

The  complexity  of  most  large-scale  systems  of  systems  is  such  that  realistic  functional 
boundaries  are  defined  to  limit  the  scope  of  the  system-of-systems  architecture.  The 
system  of  systems  architect  essentially  delegates  responsibility  for  anything  that  fits 
entirely  within  one  of  these  functional  boundaries.  The  boundaries  defined  by  the 
architecture  indicate  the  architecture's  granularity.  Attempting  to  micro  manage  the 
solution  by  defining  large  numbers  of  very  small  granules  is  fraught  with  risk. 
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The  same  principle  applies  to  the  architecture  of  a  software  application.  An  appropriate 
granularity  is  selected  to  represent  the  architecture  of  the  software  application,  but  the 
responsibility  for  anything  that  exists  exclusively  within  a  single  boundary  is  delegated  to 
a  software  developer.  It  may  still  be  appropriate  to  ensure  the  provision  of  functionality 
within  the  application  can  be  achieved  in  an  open  and  evolvable  manner.  Hence,  an 
openness  assessment  of  an  application  will  focus  more  on  the  infrastructures  and 
processes  defined  to  support  third-party  provision  and  integration  of  functionality  directly 
into  the  application. 


3.1  Granularity  Layers 

It  is  up  to  the  assessor  to  determine  an  appropriate  number  of  layers  to  conceptualise  the 
target  solution.  The  selection  of  names  for  each  of  these  layers  is  also  a  point  of  personal 
preference.  However,  for  the  purposes  of  examples  in  this  document,  we  have  chosen  six 
hierarchical  layers  of  granularity  named  as  follows: 

Module  —  The  smallest  unit  of  functionality  that  is  to  be  considered  as  a  black-box  within 
a  host  Application.  Module  openness  is  not  a  valid  assessment  criteria. 

Application  —  A  software  program  or  hardware  unit  that  comprises  a  collection  of 
interacting  modules  or  related  functionality  that  can  be  managed  as  a  single  entity,  if 
required.  Multiple  applications  may  be  combined  to  form  a  Subsystem. 

Subsystem  —  A  group  of  related  and  interacting  Applications  that  combine  to  form  a 
capability  unit  within  a  wider  System. 

System  —  A  group  of  Subsystems  that  combine  and  interact  with  one  another  to  offer  a 
coherent  operational  capability. 

System  of  Systems  —  A  heterogeneous  group  of  loosely  related  and  largely  independently 
evolved  Systems  that  combine  through  the  sharing  of  data  or  control  to  provide  a  multi¬ 
discipline  solution. 

Enterprise  —  The  collection  of  Systems  and  Systems  of  Systems  throughout  the 
organisation  that  share  data,  control  and/  or  component  capabilities. 
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4.  Technology  Readiness  Level 

Assessing  the  openness  of  a  system  that  is  at  a  planning  stage  requires  an  entirely  different 
focus  than  an  assessment  of  a  mature,  deployed  system.  Hence,  the  technology  readiness 
concept  (or  a  variation  thereof)  offers  a  way  to  vary  the  range  of  questions  used  to  assess 
openness  to  best  suit  the  target  system's  level  of  maturity. 


5.  Assessment  Focus 


There  are  six  assessment  categories  within  the  openness  and  evolvability  assessment. 
These  are: 

•  Business  Processes 

•  Legal  Framework 

•  Engineering  Processes 

•  Architecture 

•  Infrastructures 

•  Interfaces 

Process  characteristics  look  at  how  well  the  documented  processes  and  procedures 
support  the  system's  acquisition,  development  and  integration  activities.  Implementation 
characteristics  look  at  the  resulting  design  and  realization  of  the  system  itself. 

In  addition  to  these  six  categories,  assessment  topics  specifically  addressing  standards  and 
documentation  assessment  are  also  defined.  These  additional  topics  are  referenced  by 
several  of  the  assessment  categories.  A  synopsis  of  each  assessment  category  is  provided 
in  the  following  sections. 


5.1  Business  Process 

Focusing  primarily  on  the  acquisition  organisation  or  the  part  of  the  project  team  that  is 
tasked  to  oversee  the  acquisition  and  development  of  the  system,  the  Acquisition  Process 
focus  evaluates  risks  to  openness  associated  with  the  policies  and  procedures  that  are 
defined  for  the  acquisition  role. 

•  Identification  of  key  interfaces  —  evolution  planning 

•  Personnel  training  in  open  systems 

•  Defined  responsibilities 

•  Conflict  of  Interest  avoidance  through  Independence  between  roles 
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•  Migration  processes 

•  Compliance  to  documented  processes 

•  Periodic  Standards  Review  [Obsolescence/ Emergence] 

•  Acquisition  methodology. 

5.2  Legal  Framework 

In  order  to  successfully  utilise  multiple  component  vendors  in  an  independent  manner  (i.e. 
in  isolation  of  the  original  system  prime),  the  system  must  be  governed  by  an 
appropriately  constructed  legal  framework.  The  Legal  Framework  focus  evaluates  the 
completeness  and  sufficiency  of  such  a  framework. 


5.3  Engineering  Process 

Processes  and  policies  directly  associated  with  the  system  development  and  integration 
fall  into  the  Engineering  Process  focus.  These  include: 

•  Configuration  management 

•  Independent  Verification  and  Validation 

•  Independence  of  responsibilities 

•  Intellectual  Property  management. 

5.4  Architecture 

An  open  system  requires  a  system  architecture  to  be  well  defined  and  managed 
throughout  the  life-cycle.  The  Architecture  assessment  foci  evaluate  architectural 
characteristics  that  influence  openness  such  as: 

•  Quality  and  completeness  of  architecture  documentation 

•  Consistency  amongst  architecture  documentation 

•  Configuration  management  of  the  architecture  documentation 

•  Dissemination  techniques  in  use  (may  be  a  development  process  characteristic) 

•  Granularity  of  the  architecture  (size  of  functional  blocks) 

•  Modularity  (cohesion  and  coupling) 

•  Support  for  multiple  customers/  deployments. 
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5.5  Interfaces 

Interfaces  define  a  granule's  externally  visible  boundary  to  which  other  granules  can 
connect  to  exchange  data  and/or  instructions  for  control.  For  the  purposes  of  the 
Openness  Assessment,  an  Interface  does  not  include  the  infrastructure  necessary  to 
facilitate  the  transport  of  data  between  granules. 

•  Completeness  of  interface  definitions  (including  definitions  for  all  externally 
visible  interfaces  along  with  completeness  of  each  of  those  definitions)  —  include 
performance  constraints/ requirements,  units  of  measure,  context  metadata 

•  Availability  of  interfaces  to  third-parties 

•  Custodianship 

•  Configuration  management  of  interfaces 

•  Documentation  quality 

•  Reference  implementations/Test  Harnesses  for  use  by  third-parties 

•  Extensibility  —  ability  for  additional  subsystems  to  connect  to  interface. 

5.6  Infrastructures 

An  Infrastructure  defines  the  set  of  standards  that  combine  to  enable  the  transport  of  data 
between  interfaces. 

•  Completeness  of  infrastructure  definitions 

•  Standards 

•  Availability  of  definition  to  third-parties 

•  Custodianship 

•  Reference  implementations 

•  Performance /  Scalability  of  infrastructure 

•  Documentation  quality 

•  Configuration  Management. 
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6.  Assessment  Preparation  Steps 

6.1  Identify  Architecture 

Obtain  architecture  representations.  These  should  be  at  least  functional  and  physical 
representations.  In  relation  to  the  DoDAF  (or  similar  frameworks),  the  minimum  views 
should  be: 

•  OV-2  (Operational  Node  Connectivity  Description) 

•  OV-3  (Operational  Information  Exchange  Matrix) 

•  SV-1  (Systems  Interface  Description) 

•  TV-1  (Technical  Standards  Profile). 

Note  that  there  is  no  requirement  for  architectures  to  be  presented  using  representations 
discussed  in  DoDAF,  MoDAF,  DAF,  etc.  The  architectural  views  listed  above  are  included 
for  guidance  only.  The  primary  aim  for  gathering  architecture  representations  is  to  allow 
the  assessor  to  understand  the  architect's  intention  regarding  the  system's  granularity  and 
couplings. 


6.2  Establish  granule  boundaries 

An  architecture's  granule  boundaries  should  be  identifiable  from  the  available  architecture 
documentation.  However,  care  needs  to  be  taken  to  ensure  the  scope  of  the  documentation 
corresponds  with  the  layer  of  interest  for  the  purposes  of  assessment. 

This  assessment  uses  the  terms,  "granule"  and  "granularity"  intentionally  to  avoid 
connotations  otherwise  associated  with  commonly  used  terms  such  as  "module"  and 
"component". 

The  term,  "granule"  is  used  within  this  assessment  to  represent  a  collection  of  closely 
related  functionality,  within  which  the  architect  is  willing  to  consider  beyond  the  scope  of 
management  at  the  architectural  level.  In  other  words,  the  contents  of  a  granule  can  be 
considered  a  "black-box"  for  the  purposes  of  the  architecture.  There  should  be  no  need  to 
know  anything  about  the  internal  construction  or  workings  of  a  granule  other  than  its 
functional,  physical  and  performance  characteristics  that  affect  the  ability  for  other 
granules  to  connect  to  and  interoperate  with  the  granule. 

Well  defined  evolution  plans  will  generally  outline  the  functional  and  physical  boundaries 
that  best  align  with  the  future  goals  of  the  system.  It  is  conceivable  that  preferred  granule 
boundaries  will  not  align  with  current  granule  boundaries.  However,  internal  module 
boundaries  may  remain  the  same.  This  would  be  evident  in  a  system  containing  a 
relatively  large  granule  comprising  a  large  number  of  modules  provided  by  a  single 
vendor,  where  the  future  evolution  of  the  system  would  ideally  see  the  large  granule  be 
divided  into  several  smaller  granules. 
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Establishing  an  architecture's  granule  boundaries  results  in  a  "box  and  line"  diagram 
illustrating  groups  of  functionality  that  are  considered  individual  granules,  along  with 
indications  of  any  communications  that  occur  between  those  granules. 


6.3  Isolate  Infrastructures 

Determine  the  sets  of  standards  that  combine  to  form  each  well-bounded  shared 
infrastructure.  An  infrastructure  does  not  include  any  interfaces  or  subsystem 
components,  but  provides  the  means  to  glue  subsystems  together  so  that  they  can 
interoperate  via  their  interfaces.  Architectures  may  comprise  several  distinct 
infrastructures.  For  example,  if  part  of  the  system  of  systems  involves  the  distribution  of 
real-time  data,  the  infrastructure  used  to  facilitate  real-time  data  distribution  is  likely  to  be 
entirely  different  to  the  infrastructure  used  to  distribute  ad-hoc  client/server  database 
requests. 


7.  Assessment  Overview 


7.1  Assessment  Flowcharts 

Each  assessment  category  is  accompanied  by  at  least  one  flowchart  that  guides  the 
assessor  through  the  set  of  questions  associated  with  the  assessment  topic.  The  sequential 
flow  implied  by  each  flowchart  suggests  it  is  possible  to  short-cut  the  evaluation  process  if 
a  definite  'score'  is  awarded  without  navigating  every  question  in  the  entire  flowchart.  The 
'short-cut'  representation  is  primarily  intended  to  reduce  the  visual  complexity  of  the 
flowcharts. 

When  assessing  openness  and  evolvability,  it  is  recommended  that  an  assessment  report 
be  produced  that  identifies  the  risks  to  openness  that  are  believed  to  exist  within  the 
solution.  In  order  to  achieve  a  comprehensive  set  of  risks,  the  assessment  for  each  category 
should  not  terminate  when  a  'short-cut'  score  is  awarded. 


7.2  Scoring 

Each  of  the  six  assessment  categories  group  a  set  of  questions  that  enable  the  derivation  of 
a  category  score.  The  score  is  an  ordinal  value  that  represents  how  well  the  solution  meets 
the  associated  criteria  believed  necessary  to  achieve  an  open  and  evolvable  system. 

Each  ordinal  value  also  implies  a  level  of  risk  that  the  system  will  not  achieve  an  open  an 
evolvable  system.  Lienee,  a  colour  is  paired  with  each  ordinal  score  to  represent  the  risk 
level. 
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7.2.1  Comprehensive 

Representing  the  ultimate  target  score  for  each  assessment  category.  Comprehensive 
indicates  the  solution  has  extensively  addressed  all  known  issues  within  an  associated 
assessment  scope  that  could  negatively  affect  the  ability  for  the  solution  to  be  open  and 
evolvable.  Metrics  resulting  in  a  Comprehensive  score  represent  Negligible  Risk  to  the 
solution  being  open  and  evolvable. 


(  Comprehensive} 


7.2.2  Sufficient  C  Sufficient  J 

An  assessment  that  is  considered  Sufficient  indicates  that  the  solution  substantially 
addresses  all  of  the  known  issues  (within  the  associated  assessment  scope)  that  could 
negatively  affect  the  ability  for  the  solution  to  be  open  and  evolvable.  A  Slight  Risk  exists 
that  long  term  openness  and  evolvability  may  be  hindered  because  of  a  minor  discrepancy 
or  lack  of  process  that  could  otherwise  ensure  openness.  _ 

7.2.3  Partial  (_  Partial  j 

While  some  aspects  of  the  associated  assessment  topic  positively  contribute  to  an  open  and 
evolvable  solution.  Partial  scores  indicate  that  at  least  one  aspect  introduces  a  Substantial 
Risk  that  openness  will  not  be  achieved.  A  large  number  of  openness  and  evolvability 
assessment  metrics  result  in  a  Partial  score.  However,  the  level  of  effort  required  to  resolve 
many  of  the  offending  issues  can  often  be  small  —  particularly  if  other  aspects  of  the 
assessment  topic  are  already  adequately  addressed. 


7.2.4  Limited 


A  score  of  Limited  indicates  there  is,  at  best,  only  a  very  small  ability  for  the  solution  to 
support  independent  third-party  integration.  A  Major  Risk  exists  that  openness  and 
evolvability  will  not  be  achieved. 


7.3  Reporting 

The  elementary  result  of  the  openness  and  evolvability  assessment  is  a  set  of  six  scores  that 
indicate  the  extent  to  which  the  candidate  solution  is  considered  open  and  evolvable  in 
relation  to  the  six  major  assessment  categories.  These  scores  provide  only  a  coarse 
representation  of  a  solution's  openness  and  caution  should  be  exercised  before  exhibiting 
the  six  scores  in  isolation  of  supporting  evidence. 

It  is  strongly  suggested  that  any  assessment  be  reported  as  a  combination  of  the  six 
category  scores  together  with  a  synopsis  of  the  identified  risks  (together  with  their  impact 
on  the  overall  openness  score). 
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